Automatic segmentation process of a 3D medical image by one or several neural networks through structured convolution according to the anatomic geometry of the 3D medical image

ABSTRACT

This invention concerns an automatic segmentation method of a medical image making use of a knowledge database containing information about the anatomical and pathological structures or instruments, that can be seen in a 3D medical image of a×b×n dimension, i.e. composed of n different 2D images each of a×b dimension. 
     Said method being characterised in that it mainly comprises three process steps, namely: 
     a first step consisting in extracting from said medical image nine sub-images ( 1  to  9 ) of a/2×b/2×n dimensions, i.e. nine partially overlapping a/2×b/2 sub-images from each 2D image; 
     a second step consisting in nine convolutional neural networks (CNNs) analysing and segmenting each one of these nine sub-images ( 1  to  9 ) of each 2D image; 
     a third step consisting in combining the results of the nine analyses and segmentations of the n different 2D images, and therefore of the nine segmented sub-images with a/2×b/2×n dimensions, into a single image with a×b×n dimension, corresponding to a single segmentation of the initial medical image.

RELATED APPLICATION

This application is a National Phase of PCT/EP2019/050553 filed on Jan.10, 2019 which claims the benefit of priority from U.S. ProvisionalPatent Application No. 62/615,525, filed on Jan. 10, 2018, the entiretyof which are incorporated by reference.

BACKGROUND Field of the Invention

The present invention is related to the field of data processing, morespecifically to the treatment and analysis of images, in particular thesegmentation of medical images, and concerns an automatic segmentationprocess of a 3D medical image by one or several neural networks throughstructured convolution according to the anatomic geometry of the 3Dmedical image.

Description of the Related Art

A three-dimensional image made from a medical imaging device such as ascanner, MRI, ultrasound, CT or SPEC type image is composed of a set ofvoxels, which are the basic units of a 3D image. The voxel is the 3Dextension of the pixel, which is the basic unit of a 2D image. Eachvoxel is associated with a grey level or density, which can beconsidered to be the result of a 2D function F(x, y) or a 3D functionF(x, y, z), where x, y and z denote spatial coordinates (see FIG. 1).

The views of FIG. 2 illustrate the definition of a 3D medical imagesegmentation as per a transverse view.

Typically, a 2D or 3D medical image contains a set of anatomical andpathological structures (organs, bones, tissues, . . . ) or artificialelements (stents, implants, . . . ) that clinicians have to delineate inorder to evaluate the situation and to define and plan their therapeuticstrategy. In this respect, organs and pathologies have to be identifiedin the image, which means labelling each pixel of a 2D image or eachvoxel of a 3D image. This process is called segmentation.

In case of CT and MRI images acquired in clinical routine, they can beconsidered as a series of n rectangular or square 2D images (along the Zaxis) with a×b dimension (along the X and Y axis). In general, they havea standard dimension along the X and Y axis equal to 512×512 pixels,which means that the dimensions of the transversal plane are usuallya=b=512. By contrast, the number n of slices (dimension along the Zaxis) is in turn highly variable and depends on the dimension of theobserved region.

It can therefore be envisaged to analyse the transversal plane as awhole or to divide it into smaller, for example four, 256×256sub-images, hence being faster to analyse separately. The foursub-images that have been created cover all the voxels of the initialimage and their dimension is a/2×b/2×n.

FIG. 3A illustrates the three main planes to be considered when takingmedical images of a human subject and FIG. 3B illustrates the divisionof a slice along the transversal plane of the medical image into four256×256 sub-images.

There are many methods to make a segmentation. Among these methods,neural networks are part of the category dedicated to artificialintelligence algorithms and have the benefit of being automatic.

There are many variations of these algorithms, but they remain oftenlimited to a standard architectural basis that is non-specific tomedical imaging, and in particular non-specific to its content.

Image content in medical imaging is however very recurrent, especiallyin CT and MRI images. In the centre of the image we systematically havethe patient surrounded by air except underneath him/her where there isthe operating table (on which the patient is usually lying during theimaging procedure).

Thus, unlike a photographic image, where the environment changes fromone photo to another, the medical image is as structured and formattedas an ID picture for a passport: environment and position are always thesame, only details of the person's face change.

In the case of a medical image of the thorax, ribs will for instancealways be connected to the spine at the back and to the sternum in thefront, encompassing both lungs, between which lies the heart. Of coursethere can be variations such as inverted lung position or a missinglung, but these instances occur very infrequently compared to normalanatomical variation. As for the other areas (head, abdomen, pelvis,upper or lower limbs), the same observation can be made and the sameprinciple applied.

The images of FIG. 4 illustrate, by way of three examples, how variousanatomical areas (thorax, abdomen and pelvis) show a regulardistribution of the relative organ localization.

Based on these findings, the inventors have acknowledged that in thiscontext the aforementioned sub-image division takes on a differentmeaning because it becomes possible to use this division to locatestructures within the sub-images that are not found in the othersub-images.

For example, as can be seen on FIG. 5, sub-image division (here 256×256sub-images of a 512×512 transverse image of an abdomen) can contain avery regular anatomical structure, that can make the associated networkto be developed more robust, more efficient and more specialized.

FIG. 5 illustrates how various partitioning of and sub-image extractionfrom a same transverse image of the abdomen allows to systematicallylocate recurrent anatomical structures in the same regions.

In the first image of FIG. 5 for instance, the gallbladder will veryoften be found in the upper left sub-image, the right kidney in thelower left sub-image and the left kidney in the lower right sub-image.The spine will systematically belong to the sub-image identified inimage 2. The liver will systematically be part of the left sub-images ofimage 1 or 3, whereas the spleen will be in the right sub-images.Finally, aorta and vena cava will be together in the sub-image of image4, but separated in the sub-images of image 3, vena cava being in theleft one and aorta in the right one.

Objects and Summary

Thus, the basic idea of the invention is to make use of several specificsub-image divisions allowing to systematically locate recurrentanatomical structures in the same regions and to exploit and combine theinformation collected from separate analyses of these sub-images, inorder to develop a new analysis procedure of medical images usingconvolutional neural networks (CNNs) exploiting the specificlocalization information of organs.

Therefore, the present invention concerns as it main object an automaticsegmentation method of a medical image making use of a knowledgedatabase containing information about the anatomical and pathologicalstructures or instruments, that can be seen in a 3D medical image ofa×b×n dimension, i.e. composed of n different 2D images each of a×bdimension,

method characterised in that it mainly comprises the following threeprocess steps, namely:

a first step consisting in extracting from said medical image ninesub-images of a/2×b/2×n dimensions, i.e. nine partially overlappinga/2×b/2 sub-images from each 2D image;

a second step consisting in nine convolutional neural networks analysingand segmenting each one of these nine sub-images of each 2D image;

a third step consisting in combining the results of the nine analysesand segmentations of the n different 2D images, and therefore of thenine segmented sub-images with a/2×b/2×n dimensions, into a single imagewith a×b×n dimension, corresponding to a single segmentation of theinitial medical image.

More precisely, the invention proposes an automatic segmentation processafter a knowledge database has learned the anatomical and pathologicalstructures, or instruments that can be seen in the 3D medical image ofa×b×n dimension, via an algorithm composed of three steps. The firststep consists in extracting nine sub-images of a/2×b/2×n dimensions, thesecond step consists in nine Convolutional Neural Networks (CNNs)analysing one of these nine sub-images and the third step consists incombining the results of the nine analyses, and therefore of the ninesegmented sub-images with a/2×b/2×n dimensions, into a single image witha×b×n dimension. The output is a single segmentation of the initialimage. The main originality lies in this global architecture and in thepartitioning of the original image analysis based on CNN in ninesub-image analyses based on CNN.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be better understood using the description below,which relates to at least one preferred embodiment, given by way ofnon-limiting example and explained with reference to the accompanyingdrawings, wherein:

FIG. 1 illustrates a scanning machine, an image of a related 2D imagegrid and a related 3D image grid;

FIG. 2 illustrates the definition of a 3D medical image segmentation asper a transverse view;

FIG. 3A and 3B illustrate the three main planes to be considered whentaking medical images of a human subject (FIG. 3A) and illustrates thedivision of a slice along the transversal plane of the medical imageinto four 256 x 256 sub-images (FIG. 3B);

FIG. 4 illustrate, by way of three examples, how various anatomicalareas (thorax, abdomen and pelvis) show a regular distribution of therelative organ localization;

FIG. 5 illustrates how various partitioning of and sub-image extractionfrom a same transverse image of the abdomen allows to systematicallylocate recurrent anatomical structures in the same regions;

FIG. 6 is a schematical representation illustrating graphically theprocessing steps of the inventive method, namely: the specific imagedivision resulting in the extraction of nine (numbered 1 to 9)sub-images from the initial a×b medical image; the analysis andsegmentation of each sub-image by a dedicated CNN (row of round spots onFIG. 6); and the multiple partial overlapping of the nine sub-imagesfrom the initial image partition and merging of the analyses results ofthe CNNs, with the definition and the grouping of sixteen complementaryfractional regions (designated A to P);

FIG. 7 illustrates by way of example a sub-set of four different imagesgenerated from the first sub-image of the example illustrated in FIG. 6(sub-image numbered 1 in FIG. 6) by means of a shifting (translations)of one pixel (or one voxel) in the three possible directions;

FIG. 8 illustrates by way of example a sub-set of nine different imagesgenerated from the same sub-image as FIG. 7, by means of a shifting(translations) of one or two pixel(s) (or voxel-s-);

FIG. 9 is a schematical representation illustrating graphically thesteps involved with the processing (segmentation) of one 2D image (oneslice of a 3D image) by a group of nine coordinated CNNs, each onededicated to the segmentation of one of the nine sub-images (1 to 9)extracted from the initial image, the individual segmentation results ofall sub-images being combined or merged into a single initial imagesegmentation;

FIG. 10 is a schematical representation, similar to that of FIG. 9,illustrating graphically the steps involved with the processing(segmentation) of a set of n (here n=5) 2D images (set of n slices of a3D image), resulting in a single image segmentation.

DETAILED DESCRIPTION

As illustrated schematically on FIGS. 6, 9 and 10 in particular, theinvention concerns an automatic segmentation method of a medical imagemaking use of a knowledge database containing information about theanatomical and pathological structures or instruments, that can be seenin a 3D medical image of a×b×n dimension, i.e. composed of n different2D images each of a×b dimension,

method characterised in that it mainly comprises three process steps,namely:

a first step consisting in extracting from said medical image ninesub-images (1 to 9) of a/2×b/2×n dimensions, i.e. nine partiallyoverlapping a/2×b/2 sub-images from each 2D image;

a second step consisting in nine convolutional neural networks (CNNs)analysing and segmenting each one of these nine sub-images (1 to 9) ofeach 2D image;

a third step consisting in combining the results of the nine analysesand segmentations of the n different 2D images, and therefore of thenine segmented sub-images with a/2×b/2×n dimensions, into a single imagewith a×b×n dimension, corresponding to a single segmentation of theinitial medical image.

By providing a specific partitioning of the medical image to be treated,combined with a parallel processing by means of an adapted architectureof dedicated CNNs, exploiting the specific localisation information oforgans, tissues, objects and possible similar internal features, theinvention allows a faster, more accurate and more efficient segmentationof the medical image.

Typically, a known CNN algorithm which may be used within the method andthe system of the present invention is “U-Net” (see for example: “U-Net:Convolutional Networks for Biomedical Image Segmentation”; O.Ronneberger et al.; MICCAI 2015, Part III, LNCS 3951, pp 234-‘’241,Springer IPS).

“U-Net” may be implemented in connection with other known architecturessuch as “ResNet” or “DenseNet”.

The combining or merging step of the results provided by the CNNs (inparticular by two or three different CNNs in the overlapping regions ofthe sub-images) can be performed by (weighted) summing of theclassifiers, multiplication (product) or a similar adapted predictionensembling operation known to the person skilled in the art.

According to an important feature of the invention, which appearsclearly and unambiguously in FIGS. 6, 9 and 10, the nine sub-images 1 to9 of a/2×b/2 dimension each, are extracted as follows from a 2D image ofa×b dimension:

symmetrical partition of the 2D image into four sub-images 1 to 4 by themediators of the two pairs of opposed sides;

forming two sub-images 5, 6 and 7, 8 having one side in common betweenthem and centered towards the perpendicular sides of the 2D image,according to each of the two directions of said image;

forming a sub-image 9 which is centered towards the 2D image and has itssides parallel to the sides of said image.

As also shown on the aforementioned FIGS. 6, 9 and 10, the overlappingof the nine sub-images 1 to 9 is configured so as to generate sixteenfractional, complementary regions A to P of a/4×b/4 dimension each,covering together the complete surface of the considered initial 2Dimage.

In order to increase the learning speed of the knowledge database, bymaking use of the medical image structuration and contents, theinvention method may also consist in:

building a knowledge database from K segmented medical images ofa×b×N(i) dimensions, N(i) being the number of slices along Z of theimage i, i varying from 1 to K,

creating from each image of the knowledge database nine sub-sets ofimages of a/2×b/2×N(i) dimensions,

allowing for the segmentation of the nine sub-images of a/2×b/2×N(i)dimensions and for the image creation of each sub-set from the ninesub-images, and then shifting this selection by 1 to T voxel(s) in the Xand the Y directions, therefore providing nine sub-sets of 4 to (T+1)²images, each one with the same dimensions.

According to a first embodiment of the invention, shown in FIG. 9, theautomatic segmentation method consists, by means of nine 2D CNNs,

in analysing each one of the nine sub-images 1 to 9 by means of onededicated 2D CNN and by segmenting one after the other the n slices witha/2×b/2 dimensions, and then

in combining the results provided by all nine CNNs, so as to provide bysaid results merging a single initial image segmentation.

According to a second embodiment of the invention, shown in FIG. 10, theautomatic segmentation method consists, by means of nine 3D CNNs,

in analysing each one of the nine sub-images by means of one dedicated3D CNN and by segmenting one after the other all the sub-sets of Lsuccessive slices with a/2×b/2 dimension, L ranging from 2 to n, thenumber of sub-sets of 3D sub-images with a/2×b/2 dimensions varyingbetween 1 and n−L+1, and then

in combining the analysis results provided by all nine CNNs, so as toprovide by said result merging a single initial image segmentation.

The invention also encompasses a system for performing an automaticsegmentation of a medical image by implementing the method according toanyone of claims 1 to 6, characterised in that it comprises at least onecomputer device hosting and allowing the coordinated operation of nineconvolutional neural networks (CNNs) adapted to perform the segmentationof at least a part of a medical image, by using information from aknowledge database, said at least one computer device also hosting andrunning programs carrying out the partitioning of medical images and themerging of partial segmentation results provided by the different CNNs.

Of course, the invention is not limited to the two embodiments describedand represented in the accompanying drawings. Modifications remainpossible, particularly from the viewpoint of the composition of thevarious elements or by substitution of technical equivalents withoutthereby exceeding the field of protection of the invention.

The invention claimed is:
 1. An automatic segmentation method of amedical image making use of a knowledge database containing informationabout the anatomical and pathological structures or instruments, thatcan be seen in a 3D medical image of a×b×n dimension, i.e. composed of ndifferent 2D images each of a×b dimension, wherein said methodcomprises: three process steps, namely: a first step of extracting fromsaid medical image nine sub-images (1 to 9) of a/2×b/2×n dimensions,i.e. nine partially overlapping a/2×b/2 sub-images from each 2D image; asecond step of, in nine convolutional neural networks (CNNs), analyzingand segmenting each one of these nine sub-images (1 to 9) of each 2Dimage; a third step of combining the results of the nine analyses andsegmentations of the n different 2D images, and therefore of the ninesegmented sub-images with a/2×b/2×n dimensions, into a single image witha×b×n dimension, corresponding to a single segmentation of the initialmedical image; wherein the nine sub-images (1 to 9) of a/2×b/2 dimensioneach, are extracted as follows from a 2D image of a×b dimension:symmetrical partition of the 2D image into four sub-images (1 to 4) bymediators of the two pairs of opposed sides; forming two sub-images (5,6 and 7, 8) having one side in common between them and centered towardsthe perpendicular sides of the 2D image, according to each of the twodirections of said image; forming a sub-image (9) which is centeredtowards the 2D image and has its sides parallel to the sides of saidimage.
 2. The automatic segmentation method according to claim 1,wherein the overlapping of the nine sub-images (1 to 9) is configured soas to generate sixteen fractional, complementary regions (A to P) ofa/4×b/4 dimension each, covering together the complete surface of theconsidered initial 2D image.
 3. The automatic segmentation methodaccording to claim 1, wherein said method further comprises: building aknowledge database from K segmented medical images of a×b×N(i)dimensions, N(i) being the number of slices along Z of the image i, ivarying from 1 to K, creating from each image of the knowledge databasenine sub-sets of images of a/2×b/2×N(i) dimensions, allowing for thesegmentation of the nine sub-images of a/2×b/2×N(i) dimensions and forthe image creation of each sub-set from the nine sub-images, and thenshifting this selection by 1 to T voxel(s) in the X and the Ydirections, therefore providing nine sub-sets of 4 to (T+1)² images,each one with the same dimensions.
 4. The automatic segmentation methodaccording to claim 1, wherein said method further comprises, by means ofnine 2D CNNs, analyzing each one of the nine sub-images (1 to 9) bymeans of one dedicated 2D CNN and by segmenting one after the other then slices with a/2×b/2 dimensions, and then combining the resultsprovided by all nine CNNs, so as to provide by said results merging asingle initial image segmentation.
 5. The automatic segmentation methodaccording to claim 1, wherein said method comprises, by means of nine 3DCNNs, analysing each one of the nine sub-images by means of onededicated 3D CNN and by segmenting one after the other all the sub-setsof L successive slices with a/2×b/2 dimension, L ranging from 2 to n,the number of sub-sets of 3D sub-images with a/2×b/2 dimensions varyingbetween 1 and n−L+1, and then combining the analysis results provided byall nine CNNs, so as to provide by said result merging a single initialimage segmentation.
 6. A system for performing an automatic segmentationof a medical image by implementing the method according to claim 1,wherein said system comprises at least one computer device hosting andallowing the coordinated operation of nine convolutional neural networks(CNNs) adapted to perform the segmentation of at least a part of amedical image, by using information from a knowledge database, said atleast one computer device also hosting and running programs carrying outthe partitioning of medical images and the merging of partialsegmentation results provided by the different CNNs.